added ignore_case for Snowflake #308
Conversation
|
@dataders need you help with this integration test failure, its not related to any of the changes i made. |
|
@cakkinep - I'll have a look to see what's up with those CI errors. |
|
Some reason some of those |
|
Yeah, saw that in the errors, couldn't do much as I didn't have access to the CI environment and resources it is running against. Thanks for working on the fix. |
|
@robby-rob can you please check if this PR will solve for what you were looking for in #289 ? |
|
@jeremyyeo , @dataders , tagging you both for a review to get this merged. Please let me know if i may have missed anything . |
|
@jeremyyeo please let me know if you are not the right person to request a review and get this MR move forward. |
This appears to solve what I was working on. Is there a case for allowing the ignore_case option when not using the infer_schema option? I believe I added that in when #257 was merged, but not sure how important it is. |
@robby-rob-slalom Thanks for your feedback. |
|
|
||
| {% if infer_schema %} | ||
| {% set query_infer_schema %} | ||
| select * from table( infer_schema( location=>'{{external.location}}', file_format=>'{{external.file_format}}') ) | ||
| select * from table( infer_schema( location=>'{{external.location}}', file_format=>'{{external.file_format}}', ignore_case=>true) ) |
There was a problem hiding this comment.
May want to have the ignore_case option here use the ignore_case variable, defaulting to blank or false to match Snowflake defaults.
Something similar to this commit
There was a problem hiding this comment.
I am not using column name from the results of this ignore_case=>true, instead i am explicitly using GET_IGNORE_CASE function on the inferred columns to prevent any issues , my observation is when using ignore_case=>true snowflake is converting the columns to upper case by default ( snowflake's interpretation of ignoring case) and using them in the value:: reference, this would not get the correct value.
By explicitly using the function , i am able to make it truly ignore case.
corresponding code on line 59 here - https://github.com/dbt-labs/dbt-external-tables/pull/308/files#diff-bab29748b029dbac246005b734e423bdc39eb7735844083cd6902e1afbcab7f7R59
|
@dataders , @jeremyyeo , can you please look into this or get attention of someone who can look into merging this MR? |
|
@jeremyyeo , @dataders , can you please review and get this merged? |
|
@jeremyyeo , @dataders if you are not the right people to tag for getting this reviewed and merged, please tag those who are responsible. |
|
How can we get this merged? @dataders @jeremyyeo |
|
@jeremyyeo can you please review and let me know if anything else needs to be done to get this merged? |
Description & motivation
Snowflake external tables by default do not ignore case of columns of underlying external File, this poses a challenge when defining to exactly define the column names to be able to successfully create external tables and have the column values populated.
This PR adds the capability to use GET_IGNORE_CASE function to lookup the column value instead of standard
value:reference.Checklist